Control apparatus, object detection system, object detection method and program

ABSTRACT

A control apparatus capable of efficiently detecting a target object even when the target object is shielded by other objects is provided. An object recognition unit 114 recognizes a target object 80 present in a 3D environment 4 by using measurement data acquired from a sensor 12. An information generation unit 116 generates 3D environmental information by integrating a plurality of measurement data. A position determination unit 120 determines an optimal position of the sensor 12 for performing the next measurement. A sensor control unit 140 moves the sensor 12 to the determined optimal position. The position determination unit 120 determines, by using the 3D environmental information, a position of the sensor 12 where the sensor 12 can take an image in which a size of an area shielded by at least one first object is larger as the optimal position.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 16/358,988, filed on Mar. 20, 2019, which claimsthe benefit of priority from Japanese patent application No.2018-079235, filed on Apr. 17, 2018. The benefit of priority is claimedto each of the foregoing, and the entire contents of each of theforegoing are incorporated herein by reference.

BACKGROUND

The present disclosure relates to a control apparatus, an objectdetection system, an object detection method and a program. Inparticular, the present disclosure relates to a control apparatus, anobject detection system, an object detection method and a program fordetecting an object by controlling a sensor.

There is a technique for detecting a target object, which is an objectto be detected, by operating a sensor such as a range sensor. In suchtechniques, it is necessary to consider that the target object may beshielded by other objects (obstacles). In relation to this technique,Japanese Unexamined Patent Application Publication No. 2015-190818discloses a work support system for improving efficiency of work forcompletion of the whole work. The work support system disclosed inJapanese Unexamined Patent Application Publication No. 2015-190818includes a measurement apparatus that measures (i.e., obtains)three-dimensional (3D) shape data of an object to be measured, atransfer apparatus that moves at least one of the objects to be measuredand the measurement apparatus and thereby changes a measurement positionwhere the measurement apparatus measures the object to be measured, anda work support apparatus that controls the transfer apparatus. Acandidate position setting unit sets, on a surface of the object to bemeasured, candidate measurement positions for the entire area of ameasurement target range designated as a measurement target range. Anavailable surface calculation unit calculates, for each of the setmeasurement positions, the number of measurable surfaces of the objectto be measured or an area (i.e., a size) of the measurable surfaces. Aranking determination unit determines, for each measurement direction, apriority order of measurement according to the calculated number or thearea (i.e., the size) of the surfaces. A measurement control unitinstructs the transfer apparatus to perform measurement in eachmeasurement direction according to the determined priority order.

SUMMARY

The present inventors have found the following problem. Depending on anoperating environment of a sensor, a plurality of target objects, whichare objects to be detected, may exist in the operating environment.Further, objects other than the target object(s) may exist in theoperating environment. In such cases, there is a possibility that thetarget object may be shielded by other objects. Therefore, there is apossibility that even if the sensor is moved at random without takingthe positional relation between the target object and the otherobject(s) into consideration, it may take an enormous amount of time todetect the target object. That is, there is a possibility that even ifthe sensor is moved to a certain position and performs measurement atthat position, the target object may not be measured because the areashielded by the other object(s) is large. Therefore, there is apossibility that the time that is taken to move the sensor to thatposition and perform the measurement may be wasted. In the techniquedisclosed in Japanese Unexamined Patent Application Publication No.2015-190818, objects other than the object to be measured are not takeninto consideration. Therefore, in the technique disclosed in JapaneseUnexamined Patent Application Publication No. 2015-190818, there is apossibility that when the object to be measured is shielded by anobstacle, the object to be measured cannot be efficiently detected.

The present disclosure provides a control apparatus, an object detectionsystem, an object detection method and a program capable of efficientlydetecting a target object even when the target object is shielded byother objects.

A first exemplary aspect is a control apparatus configured to detect atarget object to be detected by controlling a sensor configured tomeasure surroundings in a three-dimensional (3D) environment, including:an object recognition unit configured to recognize the target objectpresent in the 3D environment by using measurement data acquired fromthe sensor; an information generation unit configured to generate 3Denvironmental information indicating each point on an object present inthe 3D environment by integrating a plurality of measurement data; aposition determination unit configured to determine an optimal positionof the sensor for performing next measurement; and a sensor control unitconfigured to move the sensor to the determined optimal position, inwhich the position determination unit determines, by using the 3Denvironmental information, a position of the sensor where the sensor cantake an image in which a size of an area shielded by at least one firstobject is larger as the optimal position, the at least one first objectbeing the target object recognized by the object recognition unit.

Further, another exemplary aspect is an object detection systemincluding: a sensor configured to measure surroundings in a 3Denvironment; and a control apparatus configured to detect a targetobject to be detected by controlling the sensor, in which the controlapparatus includes: an object recognition unit configured to recognizethe target object present in the 3D environment by using measurementdata acquired from the sensor; an information generation unit configuredto generate 3D environmental information indicating each point on anobject present in the 3D environment by integrating a plurality ofmeasurement data; a position determination unit configured to determinean optimal position of the sensor for performing next measurement; and asensor control unit configured to move the sensor to the determinedoptimal position, and in which the position determination unitdetermines, by using the 3D environmental information, a position of thesensor where the sensor can take an image in which a size of an areashielded by at least one first object is larger as the optimal position,the at least one first object being the target object recognized by theobject recognition unit.

Further, another exemplary aspect is an object detection method fordetecting a target object to be detected by controlling a sensorconfigured to measure surroundings in a 3D environment, including:recognizing the target object present in the 3D environment by usingmeasurement data acquired from the sensor; generating 3D environmentalinformation indicating each point on an object present in the 3Denvironment by integrating a plurality of measurement data; determining,by using the 3D environmental information, a position of the sensorwhere the sensor can take an image in which a size of an area shieldedby at least one first object is larger as an optimal position of thesensor for performing next measurement, the at least one first objectbeing the recognized target object; and moving the sensor to thedetermined optimal position.

Further, another exemplary aspect is a program for performing an objectdetection method in which a target object to be detected is detect bycontrolling a sensor configured to measure surroundings in a 3Denvironment, the program being adapted to cause a computer to perform:recognizing the target object present in the 3D environment by usingmeasurement data acquired from the sensor; generating 3D environmentalinformation indicating each point on an object present in the 3Denvironment by integrating a plurality of measurement data; determining,by using the 3D environmental information, a position of the sensorwhere the sensor can take an image in which a size of an area shieldedby at least one first object is larger as an optimal position of thesensor for performing next measurement, the at least one first objectbeing the recognized target object; and moving the sensor to thedetermined optimal position.

In the present disclosure, when a target object is detected by using thesensor, the sensor can be moved to a position where the sensor canmeasure an area that has become a blind sport (i.e., that cannot beviewed) due to the first object more appropriately. Therefore, it ispossible to reduce the number of movements of the sensor and the timerequired therefor when the target object is detected by using thesensor. Consequently, according to the present disclosure, it ispossible to efficiently detect a target object even when the targetobject is shielded by other objects.

Further, the position determination unit preferably calculates, as acandidate for the optimal position, a position of the sensor where thesensor can measure a placement available area according to a positionand a shape of a storage object in which the target object can beplaced, and selects the optimal position from the candidate, theplacement available area being an area where the target object can beplaced.

By calculating viewpoint candidates as described above, it is possibleto exclude, from the optimal position, a viewpoint position that cannotcontribute to the detection of the target object depending on theposition and the shape of the storage object. Therefore, the presentdisclosure makes it possible to efficiently detect a target object.

Further, the position determination unit preferably determines whetheror not, among candidate positions, there is an unmeasured position fromwhich the sensor can measure the area shielded by the first object buthas not performed measurement yet, and when it is determined that thereis the unmeasured position, performs a process for determining theoptimal position.

The state where there is no unmeasured position means a state where thetarget object probably cannot be detected even when measurement isfurther performed. Therefore, by determining whether or not there is anunmeasured position as in the case of the present disclosure, it ispossible to terminate the process for determining the optimal positionwhen there is no unmeasured position. Therefore, the present disclosurecan finish the process for detecting the target object withoutperforming a wasteful process.

Further, the control apparatus preferably further includes a removaldetermination unit configured to, when it is determined that there is nounmeasured position, determine the first object to be removed so thatthe area shielded by the first object can be measured.

By being configured as described above, the present disclosure can makeit possible to measure an unmeasured area, which has not been able to bemeasured, by removing the first object to be removed in the subsequentprocess, and thereby efficiently detect the unmeasured area.

Further, the control apparatus preferably further includes an armcontrol unit configured to control an arm so that the determined firstobject is removed.

By being configured as described above, the present disclosure can makeit possible to automatically remove the first object to be removed.Therefore, it is possible to efficiently measure the unmeasured area.

Further, the position determination unit preferably determines whetheror not a search of the placement available area, which is the area wherethe target object can be placed, has been completed by using the 3Denvironmental information, and when it is determined that the search hasnot been completed, performs a process for determining the optimalposition.

By being configured as described above, the present disclosure canprevent, when it is determined that the search of the storage object hasbeen completed, the process for determining the optimal position frombeing performed. Therefore, the present disclosure can prevent anunnecessary process from being performed. Therefore, the presentdisclosure can prevent or reduce an increase in the time taken to detectan object.

According to the present disclosure, it is possible to provide a controlapparatus, an object detection system, an object detection method and aprogram capable of efficiently detecting a target object even when thetarget object is shielded by other objects.

The above and other objects, features and advantages of the presentdisclosure will become more fully understood from the detaileddescription given hereinbelow and the accompanying drawings which aregiven by way of illustration only, and thus are not to be considered aslimiting the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows an object detection system according to a first embodiment;

FIG. 2 is a block diagram showing a hardware configuration of the objectdetection system according to the first embodiment;

FIG. 3 is a functional block diagram showing a configuration of acontrol apparatus according to the first embodiment;

FIG. 4 is a flowchart showing an object detection method performed bythe control apparatus according to the first embodiment;

FIG. 5 is a flowchart showing a first example of a search completiondetermination method performed by a search determination unit accordingto the first embodiment;

FIG. 6 is a diagram for explaining the first example shown in FIG. 5;

FIG. 7 is a flowchart showing a second example of the search completiondetermination method performed by the search determination unitaccording to the first embodiment;

FIG. 8 is a diagram for explaining the second example shown in FIG. 7;

FIG. 9 is a diagram for explaining a viewpoint candidate calculationmethod according to the first embodiment;

FIG. 10 is a diagram for explaining the viewpoint candidate calculationmethod according to the first embodiment;

FIG. 11 is a diagram for explaining the viewpoint candidate calculationmethod according to the first embodiment;

FIG. 12 is a diagram for explaining processes performed by an optimalviewpoint selection unit according to the first embodiment;

FIG. 13 is a diagram for explaining processes performed by the optimalviewpoint selection unit according to the first embodiment;

FIG. 14 is a diagram for explaining processes performed by the optimalviewpoint selection unit according to the first embodiment;

FIG. 15 is a flowchart showing an example of a process for determiningwhether or not there is an unmeasured viewpoint position according tothe first embodiment;

FIG. 16 is a diagram for explaining the process shown in FIG. 15;

FIG. 17 is a diagram for explaining the process shown in FIG. 15;

FIG. 18 is a diagram for explaining the process shown in FIG. 15;

FIG. 19 shows an object detection system according to a secondembodiment;

FIG. 20 is a block diagram showing a hardware configuration of theobject detection system according to the second embodiment;

FIG. 21 is a functional block diagram showing a configuration of acontrol apparatus according to the second embodiment;

FIG. 22 is a flowchart showing an object detection method performed bythe control apparatus according to the second embodiment;

FIG. 23 is a flowchart showing an example of a method for determining anobject to be removed according to the second embodiment;

FIG. 24 is a diagram for explaining the method shown in FIG. 23;

FIG. 25 is a diagram for explaining the method shown in FIG. 23;

FIG. 26 is a diagram for explaining the method shown in FIG. 23;

FIG. 27 is a diagram for explaining the method shown in FIG. 23; and

FIG. 28 is a diagram for explaining the method shown in FIG. 23.

DESCRIPTION OF EMBODIMENTS First Embodiment

Embodiments according to the present disclosure are describedhereinafter with reference to the drawings. Note that the same symbolsare assigned to substantially the same components.

FIG. 1 shows an object detection system 1 according to a firstembodiment. Further, FIG. 2 is a block diagram showing a hardwareconfiguration of the object detection system 1 according to the firstembodiment. The object detection system 1 according to the firstembodiment includes an object detection apparatus 10 and a controlapparatus 100. The object detection apparatus 10 includes a sensor 12and a driving unit 14 that drives the sensor 12.

The control apparatus 100 is, for example, a computer. The controlapparatus 100 is connected to the object detection apparatus 10 througha wired or wireless communication link 2 so that they can communicatewith each other. Therefore, the control apparatus 100 is connected tothe sensor 12 and the driving unit 14 so that they can communicate witheach other.

Note that in FIG. 1, the control apparatus 100 and the object detectionapparatus 10 are shown as physically separate apparatuses. However, thecontrol apparatus 100 may be incorporated into the object detectionapparatus 10. Further, at least one component of the control apparatus100 (which will be described later) may be incorporated into the objectdetection apparatus 10. In such a case, the object detection apparatus10 also has functions as a computer.

The object detection apparatus 10 moves in a three-dimensional (3D)environment 4. The object detection apparatus 10 can autonomously movein the 3D environment 4. Note that the 3D environment 4 may be expressedby an orthogonal coordinate system or may be expressed by a polarcoordinate system. In the following descriptions, an example in whichthe 3D environment 4 is expressed by an (X, Y, Z)-orthogonal coordinatesystem is shown.

The sensor 12 is a 3D sensor capable of measuring a distance to anobject, such as a depth sensor, a range sensor (or a distance censor),or a 3D camera (a stereo camera). The sensor 12 is, for example, a lidar(LIDAR: Light Detection and Ranging) or the like. The object detectionapparatus 10 (the sensor 12) has five degrees of freedom by the drivingunit 14 as described below.

As indicated by an arrow A, the driving unit 14 moves the objectdetection apparatus 10 (the sensor 12) in an X-axis direction of the 3Denvironment 4. Further, as indicated by an arrow B, the driving unit 14moves the object detection apparatus 10 (the sensor 12) in a Y-axisdirection of the 3D environment 4. Further, as indicted by an arrow C,the driving unit 14 moves the sensor 12 in a Z-axis direction of the 3Denvironment 4 (i.e., in a vertical direction). Further, as indicted byan arrow D, the driving unit 14 rotates (turns) the sensor 12 inparallel to an XY-plane of the 3D environment 4 (i.e., in a horizontaldirection). Further, as indicted by an arrow E, the driving unit 14rotates (swings) the sensor 12 in an up/down direction of the 3Denvironment 4. That is, as indicated by the arrows A, B and C, thesensor 12 is moved by the driving unit 14 so that its 3D positioncoordinates in the 3D environment 4 changes. Further, as indicated bythe arrows D and E, the sensor 12 is moved by the driving unit 14 sothat its posture (its orientation) in the 3D environment 4 changes. Inthe following descriptions, the “movement” of the sensor 12 includes achange in the 3D position coordinates and a change in the posture.Further, the “position” of the sensor 12 includes its 3D positioncoordinates and its posture.

The sensor 12 measures surroundings of the object detection apparatus10. The sensor 12 acquires an image(s) of an object(s) present in themeasured surroundings. Further, the sensor 12 measures a distance toeach point on the object observed from the sensor 12 (the objectdetection apparatus 10). Then, the sensor 12 generates distance dataindicating the measured distance. That is, the distance data correspondsto the measurement data generated by the sensor 12. The sensor 12generates distance image data indicating a distance image (a pointcloud) as the distance data. That is, the distance data represents agroup of points (hereinafter also referred to as a point group) on thesurface of each object present around the sensor 12 (the objectdetection apparatus 10) in three dimensions. The sensor 12 scans itssurroundings with laser light (i.e., emits laser light to itssurroundings), receives reflected light reflected on an object, andcalculates a distance to the object from, for example, a differencebetween a transmission time of the laser light and a reception time ofthe reflected light. Then, the object detection apparatus 10 (the sensor12) calculates 3D coordinates (X, Y, Z) of a point at which the laserlight is reflected based on 3D position coordinates of the sensor 12 inthe 3D environment 4, an emitting direction of the laser light, and thedistance to the object. In this way, the object detection apparatus 10(the sensor 12) measures a position of each object in the 3D environment4. Note that, in the following descriptions, the term “image” also means“image data representing an image” as data to be processed ininformation processing.

A plurality of target objects 80, which are objects to be detected bythe object detection apparatus 10, are disposed in the 3D environment 4.Further, at least one storage object 90 is provided in the 3Denvironment 4. The storage object 90 includes at least one shelf board92 and a wall surface(s) 94. The storage object 90 can house a pluralityof target objects 80. In the example shown in FIG. 1, target objects 80Ato 80E are disposed (i.e., placed) in the storage object 90.

The control apparatus 100 includes, as a main hardware configuration, aCPU (Central Processing Unit) 102, a ROM (Read Only Memory) 104, a RAM(Random Access Memory) 106, and an interface unit 108 (IF; Interface).The CPU 102, the ROM 104, the RAM 106, and the interface unit 108 areconnected to each other through a data bus or the like.

The CPU 102 has functions as an arithmetic apparatus that performscontrol processes, arithmetic processes, etc. The ROM 104 has a functionof storing a control program(s), an arithmetic program(s), etc. that areexecuted by the CPU 102. The RAM 106 has a function of temporarilystoring processing data and the like. The interface unit 108 externallyreceives/outputs signals wirelessly or through a wire. Further, theinterface unit 108 receives a data input operation performed by a userand displays information for the user.

FIG. 3 is a functional block diagram showing a configuration of thecontrol apparatus 100 according to the first embodiment. The controlapparatus 100 includes an information storage unit 112, an objectrecognition unit 114, an information generation unit 116, a positiondetermination unit 120, and a sensor control unit 140 (hereinafter, alsoreferred to as “each component”). Further, the position determinationunit 120 includes a search determination unit 122 and an optimalviewpoint selection unit 124. Each component can be implemented by, forexample, having the CPU 102 execute a program(s) stored in the ROM 104.Further, necessary programs may be stored in an arbitrary nonvolatilerecording medium in advance, and may be installed as required. Note thatthe implementation of each component is not limited to theabove-described software implementation and may be implemented byhardware such as some type of circuit devices. Further, at least one ofthe aforementioned components may be implemented by physically-separateindividual hardware. This also applies to a later-described secondembodiment.

The information storage unit 112 stores 3D environmental information,storage object information, target object information, and sensorinformation. Further, the information storage unit 112 may storerecognition results obtained by the object recognition unit 114 (whichwill be described later). The “3D environmental information” isinformation indicating 3D coordinate data of each point (of a pointgroup) on each object present in the 3D environment 4. When there are aplurality of 3D environments 4, the information storage unit 112 maystore a plurality of 3D environmental information pieces. For example,3D environmental information may be generated and stored for eachenvironment such as an environment of a house, an environment of a shop,an environment of a tenant (i.e., a rental room), and an environment ofa floor.

The 3D environmental information indicates, for example, whether or notthere is any object at a given point represented by 3D coordinates (X,Y, Z). Therefore, the control apparatus 100 and the object detectionapparatus 10 can recognize a shape of an object by detecting presence ofsome object in consecutive 3D coordinates in the 3D environmentalinformation. The 3D environmental information is acquired by, forexample, having the sensor 12 scan the entire space of the 3Denvironment 4 and calculate 3D coordinates of each point on each object.That is, the 3D environmental information can be generated by using thesensor 12. Note that the 3D environmental information corresponds toinformation (or data) obtained by integrating (or combining) 3Dmeasurement data (distance image data) generated by measurementperformed by the sensor 12. That is, the 3D environmental informationcan be generated by integrating (or combining) 3D measurement dataobtained by having the sensor 12 perform measurement from one viewpointposition with 3D measurement data obtained by having the sensor 12perform measurement from another viewpoint position. In other words, the3D environmental information is successively updated as 3D measurementdata is successively generated at a plurality of viewpoint positions bythe sensor 12.

The “storage object information” is information related to the storageobject 90. The storage object information indicates a 3D shape and asize (dimensions) of the storage object 90. For example, the storageobject information may be CAD (computer-aided design) data of theobject. Further, the storage object information indicates a position (3Dcoordinates) in the 3D environment 4. Therefore, by integrating (orcombining) the storage object information into the 3D environmentalinformation, the control apparatus 100 can recognize where the storageobject 90 is located in the 3D environment 4 by using the 3Denvironmental information. In other words, the control apparatus 100 canrecognize which coordinates (X, Y, Z) in the 3D environment 4 the shelfboard(s) 92 and the wall surface(s) 94 are located at by using the 3Denvironmental information. Further, the storage object information mayalso include information indicating an area (i.e., a size) of the shelfboard 92, an edge(s) of the shelf board 92, an opening(s) in the shelfboard 92, etc.

The “target object information” is information necessary to detect thetarget object to be detected by the object detection apparatus 10. Eachtarget object 80 is registered in the object detection system 1 bystoring its target object information in the information storage unit112. The target object information may include identificationinformation (e.g., a registered name) of the target object 80. Further,for example, the target object information indicates a shape and a size(dimensions) of the target object 80. For example, the target objectinformation may be CAD data of the object. Note that the target objectinformation may not include position information such as informationindicating where the corresponding object is placed. Therefore, althoughthe control apparatus 100 can recognize that some object is placed onthe shelf board 92 by using the 3D environmental information, it cannotrecognize which target object 80 corresponds to that object unless alater-described object recognition process is performed. The objectdetection system 1 makes a search as to where a registered target object80 is located in the 3D environment 4 by using the target objectinformation and the 3D environmental information. In other words, theobject detection system 1 detects (i.e., determines) which area in the3D environment 4 corresponds to the target object 80.

The “sensor information” is information related to measurement performedby the sensor 12. For example, the sensor information indicates an angleof view (a viewing angle; a field-of-view range), a focal length, aresolution, number of pixels, and the like of the sensor 12. That is,the sensor information may indicate a measurable range of the sensor 12.In this way, a size, a resolution, and the like of 3D image data(distance image data) generated by the sensor 12 can be specified.

When the 3D measurement data (the distance image) is generated by themeasurement performed by the sensor 12, the object recognition unit 114recognizes the target object 80 present in the 3D environment 4 by usingthe 3D measurement data and information stored in the informationstorage unit 112. Specifically, the object recognition unit 114 detectsa registered target object 80 from the 3D measurement data (and, ifnecessary, the 3D environmental information). More specifically, theobject recognition unit 114 calculates, for each object, a differencebetween information indicating the shape of that object in the 3Dmeasurement data (the distance image) and object information (CAD dataor the like) indicating the shape of the target object 80. Then, theobject recognition unit 114 recognizes that an object for which thecalculated difference is smaller than a predetermined threshold as thetarget object 80. Further, the object recognition unit 114 associatesidentification information of the detected target object 80 withposition information indicating a position where that target object 80is located.

The information generation unit 116 generates 3D environmentalinformation. Specifically, when an object recognition process isperformed by the object recognition unit 114, the information generationunit 116 updates the 3D environmental information by using a result ofthe object recognition process (a recognition result). Note that the 3Denvironmental information may include only position informationcorresponding to the storage object 90 at the initial stage. In otherwords, before the sensor 12 starts measurement, only the storage object90 is present in a 3D virtual space represented by the 3D environmentalinformation. Then, every time the sensor 12 performs measurement and anobject is recognized by the object recognition unit 114, the informationgeneration unit 116 integrates 3D measurement data generated by themeasurement by the sensor 12 into the 3D environmental information. Inthis way, information on objects included in the 3D environmentalinformation increases.

It should be noted that depending on the viewpoint of the sensor 12 orthe position of the target object 80, a part of the target object 80 maybe shielded by another target object 80 or the wall surface 94 or thelike of the storage object 90 even though that target object 80 ispresent in the field of view (the angle of view) of the sensor 12. Insuch a case, the shielded part of the target object 80 is notphotographed by the sensor 12. For example, a target object 80B ispresent in front of a target object 80C in FIG. 1. Therefore, there is apossibility that, from a certain viewpoint position, a part of thetarget object 80C (i.e., a lower-right part of the target object 80Cshielded by the target object 80B) is not photographed by the sensor 12.In this case, the target object 80B acts as an obstacle and forms ablind spot for the sensor 12 when the sensor 12 measures (photographs)the target object 80C. Further, at least a part of the target object 80Cis present in the blind spot.

In this case, there is a possibility that the object 80 may not berecognized by the object recognition unit 114 due to lack of the amountof information indicating its shape and the like. In such a case, theinformation generation unit 116 adds information indicating a shape ofthe part of the target object 80, which has not been recognized,photographed by the sensor 12 in the 3D environmental information. Inthis case, although the control apparatus 100 can recognize that someobject exists in that position by using the 3D environmentalinformation, it cannot recognize which target object 80 corresponds tothat object. Therefore, the 3D environmental information may includeinformation related to a recognized target object 80 and informationrelated to a target object 80 that has not been recognized (hereinafteralso referred to as unrecognized target object 80).

The position determining unit 120 determines an optimal viewpointposition (an optimal position) of the sensor 12 for the next measurement(later-described S130 etc. in FIG. 4). Specifically, the positiondetermining unit 120 determines, as the optimal viewpoint position, aviewpoint position from which it is expected that the target object 80,which is possibly disposed in the storage object 90 but has not beenrecognized, can probably be recognized when the sensor 12 is moved tothat viewpoint position and measures (photographs) the that targetobject 80 the next time. Here, the recognized object 80 (e.g., thetarget object 80B in FIG. 1) is referred to as a first object and thetarget object 80 that has not been recognized because it is partiallyshielded by the first object (e.g., the target object 80C in FIG. 1) isreferred to as a second object. In this case, the position determiningunit 120 determines, as the optimal viewpoint position, a viewpointposition from which the sensor 12 can measure an area that is shieldedby the first object and becomes a blind spot by using the 3Denvironmental information. That is, in order to enable an area shieldedby at least one first object recognized by the object recognizing unit114 to be measured by the sensor 12, the position determining unit 120determines, as the optimal viewpoint position, a position of the sensor12 where the sensor 12 will perform the next measurement according tothe position of the first object by using the 3D environmentalinformation. In other words, the position determination unit 120determines, by using the 3D environmental information, a position of thesensor 12 where the sensor 12 can take a distance image in which a sizeof an area shielded by at least one first object recognized by theobject recognition unit 114 is larger as the optimal viewpoint position.In this way, the control apparatus 100 can move the sensor 12 to aposition where the sensor 12 can measure the area, which is shielded bythe first object and becomes a blind spot, more appropriately.Therefore, the control apparatus 100 according to the first embodimentcan efficiently detect the target object 80 even when the second objectis shielded by the first object.

The sensor control unit 140 moves the sensor 12 to the optimal viewpointposition determined by the position determination unit 120. Then, thesensor control unit 140 controls the sensor 12 so as to performmeasurement at the viewpoint position to which the sensor 12 has moved(i.e., at the optimal viewpoint position). Then, the sensor control unit140 acquires a distance image (3D measurement data) generated by thesensor 12. Further, the sensor control unit 140 outputs the 3Dmeasurement data to the object recognition unit 114.

The search determination unit 122 determines whether or not a search ofthe storage object 90 has been completed (later-described S110 to S112in FIG. 4). Then, when the search determination unit 122 determines thatthe search of the storage object 90 has been completed, a recognitionresult for the target object 80 is output to, for example, the interfaceunit 108 or the like. On the other hand, when the search determinationunit 122 determines that the search of the storage object 90 has notbeen completed, later-described processes performed by the optimalviewpoint selection unit 124 are performed. Specifically, the searchdetermination unit 122 determines whether or not a search of an areawhere the target object 80 can be placed (a placement available area) inthe storage object 90 has been completed by using the 3D environmentalinformation. Details will be described later. Note that the placementavailable area may be a flat surface (e.g., the top surface of the shelfboard 92) on which the target object 80 can be placed, or a space (e.g.,a space between upper and lower shelf boards 92) in which the targetobject 80 can be disposed.

Note that when the search of the storage object 90 by the sensor 12 hasbeen completed, there is a high possibility that no target object 80that has not been recognized exists in the storage object 90 (i.e., allthe target objects 80 disposed in the storage object 90 have beenrecognized). That is, there is a high possibility that the measurementfor detecting target objects 80 has been sufficiently performed.Therefore, performing further measurement by the sensor 12 is probablywasteful. Therefore, it is possible to prevent an unnecessary processfrom being performed by preventing the optimal viewpoint selection unit124 from performing a process when the search determination unit 122determines that the search of the storage object 90 has been completedas described in the first embodiment. Therefore, the control apparatus100 according to the first embodiment can prevent or reduce an increasein the time taken to detect an object.

The optimal viewpoint selection unit 124 calculates viewpoint positionsof the sensor 12 where the sensor 12 can measure the placement availablearea as candidates for the optimal viewpoint position (viewpointcandidates) according to the position and the shape of the storageobject 90 in which the target object 80 can be disposed (later-describedS122 in FIG. 4). For example, viewpoint positions from which the shelfboard 92 of the storage object 90 cannot be measured are excluded fromthe viewpoint candidates. Further, the optimal viewpoint selection unit124 selects the optimal viewpoint position from the viewpointcandidates. Details will be described later. As described above, bycalculating viewpoint candidates, it is possible to exclude viewpointpositions that cannot contribute to the detection of the target object80 depending on the position and the shape of the storage object 90 fromthe optimal viewpoint positions. Therefore, the control apparatus 100according to the first embodiment can efficiently detect the targetobject 80.

Further, the optimal viewpoint selection unit 124 determines whether ornot, among the viewpoint candidates, there is an unmeasured viewpointposition (an unmeasured position), i.e., a viewpoint position from whichthe sensor 12 can measure an area shielded by the recognized targetobject 80 (the first object) but has not performed measurement yet(later-described S150 in FIG. 4). Then, when the optimal viewpointselection unit 124 determines that there is an unmeasured viewpointposition, it performs a process for determining an optimal viewpointposition. Details will be described later. The state where there is nounmeasured viewpoint position means a state where the target object 80cannot be detected even when measurement is further performed.Therefore, by determining whether or not there is an unmeasuredviewpoint position as described above, it is possible to terminate theprocess for selecting the optimal viewpoint position when there is nounmeasured viewpoint position. As a result, it is possible to improvethe efficiency of the detection of the target object 80. That is, it ispossible to finish the process for detecting target objects withoutperforming a wasteful process.

FIG. 4 is a flowchart showing an object detection method performed bythe control apparatus 100 according to the first embodiment. Firstly,the sensor control unit 140 controls the sensor 12 so as to performmeasurement at the current viewpoint position, and acquires 3Dmeasurement data (a distance image) from the sensor 12 (step S100).Then, as described above, the object recognition unit 114 performs anobject recognition process by using the 3D measurement data andinformation stored in the information storage unit 112 (i.e., targetobject information, 3D environmental information, storage objectinformation, etc.) (step S102). The information generation unit 116updates the 3D environmental information by using the recognition resultand the 3D measurement data (step S104).

In the example shown in FIG. 1, the target objects 80A and 80B arelocated relatively close to the front of the shelf board 92. Therefore,roughly the entire shape of each of them can be measured. Therefore, theobject recognition unit 114 can recognize the target objects 80A and80B. Then, the information generation unit 116 may add positioninformation indicating the shape of the target object 80A in an areacorresponding to the position where the target object 80A is disposed ina 3D virtual space represented by the 3D environmental information byusing the target object information of the target object 80A. Theinformation generation unit 116 may also perform a similar process forthe target object 80B.

In contrast, the object 80C is located behind the target object 80B andhence a part of its shape cannot be measured. Therefore, there is apossibility that the object recognition unit 114 cannot recognize thetarget object 80C. In this case, the information generation unit 116 mayadd position information indicating a measured part of the target object80C in an area corresponding to the position where the target object 80Cis disposed in the 3D virtual space represented by the 3D environmentalinformation by using the 3D measurement data.

Next, the search determination unit 122 determines whether or not thesearch of the storage object 90 has been completed (step S110).Specifically, the search determination unit 122 determines, by alater-described method, whether or not the measurement of the placementavailable area by the sensor 12 has been completed by using the 3Denvironmental information and the storage object information. Therefore,the process in the step S110 is performed in the 3D virtual spacerepresented by the 3D environmental information.

When it is determined that the search of the storage object 90 has notbeen completed (No at S110), the search determination unit 122determines that it is necessary to perform measurement from a viewpointposition from which measurement has not been performed. Therefore, inthis case, the search determination unit 122 outputs a signal indicatingthat further measurement is necessary to the optimal viewpoint selectionunit 124. As a result, the optimal viewpoint selection unit 124 performsa later-described process in a step S120.

On the other hand, when it is determined that the search of the storageobject 90 has been completed (Yes at S110), the search determinationunit 122 determines that at least a part of every target object 80(e.g., an upper half, a left half, etc. of every target object 80)disposed in the placement available area has been measured. Therefore,the search determination unit 122 determines whether or not labeling hasbeen performed for every target object 80 disposed in the storage object90 (step S112). Note that the labeling means, as a result of recognitionof a target objects 80 disposed in the storage object 90, associatingidentification information corresponding to that target object 80 withan area where that target object 80 is disposed.

Note that when roughly the entire shape of the target object 80 has beenmeasured in the 3D measurement data (the 3D environmental information)(i.e., when roughly the entire image of the target object 80 is includedin the distance image), labeling can be made by using the target objectinformation. On the other hand, when the entire shape of the targetobject 80 has not been measured in the 3D measurement data (the 3Denvironmental information) (i.e., when a part of an image of the targetobject 80 is missing in the distance image and information necessary forthe object recognition is insufficient), there is a possibility that theobject recognition will end in failure even when the target objectinformation is used. In such a case, labeling cannot be made.

When it is determined that labeling has been made for all the targetobjects 80 disposed in the storage object 90 (Yes at S112), the controlapparatus 100 determines that all the target objects 80 disposed in thestorage object 90 have been detected. Therefore, the control apparatus100 outputs a recognition result to the interface unit 108 (step S114).Note that the recognition result is information indicating where eachtarget object 80 is located. In other words, the recognition result isinformation indicating which area is occupied by which target object 80in the 3D virtual space represented by the 3D environmental information.

When it is determined that labeling has still not been made for all thetarget objects 80 disposed in the storage object 90 (No at S112), thesearch determination unit 122 determines that further measurement isnecessary to supplement the information which is insufficient to performthe labeling. In this case, the search determination unit 122 outputs asignal indicating that further measurement is necessary to the optimalviewpoint selection unit 124. As a result, the optimal viewpointselection unit 124 performs the later-described process in the stepS120.

FIG. 5 is a flowchart showing a first example of a search completiondetermination method performed by the search determination unit 122according to the first embodiment. Further, FIG. 6 is a diagram forexplaining the first example shown in FIG. 5. FIG. 6 is a plan viewshowing a state in which target objects 80A, 80B and 80C are disposed onthe shelf board 92 of the storage object 90 as viewed from above (asviewed in a Z-axis positive direction). Further, it is assumed that thetarget objects 80A, 80B and 80C have already been recognized.

The search determination unit 122 calculates a sum total Sa of ameasured area (i.e., a measured size) of the shelf board 92 of thestorage object 90 and areas (i.e., sizes) of bottom surfaces of therecognized target objects 80 (step S112A). Note that the measured areaof the shelf board 92 can be geometrically calculated from the 3Dmeasurement data and coordinate data in the 3D environmentalinformation. Further, the areas of the bottom surfaces of the recognizedtarget objects 80 are included in their corresponding target objectinformation beforehand. In the example shown in FIG. 6, the measuredarea (i.e., the measured size) of the shelf board 92 corresponds to thearea (i.e., the size) of the shelf board 92 excluding areas that areshielded by the target objects 80A, 80B and 80C, and hence become blindspots (i.e., hatched areas) within a field-of-view range 12 a. Further,the areas (i.e., the sizes) of the bottom surfaces of the recognizedtarget objects 80 corresponds to the sum total of the bottom areas ofthe target objects 80A, 80B and 80C. Note that when the target object80C has not been recognized, the bottom area of the target object 80C isexcluded from the sum total Sa.

The search determination unit 122 determines whether or not a ratio ofthe sum total Sa to the area Sb of the shelf board 92, i.e., a ratioSa/Sb is equal to or larger than a predetermined threshold value ThA(step S114A). Note that the area Sb of the shelf board 92 can beacquired (i.e., calculated) from the storage object information. Thearea Sb may be included in the storage object information. When theratio Sa/Sb is smaller than the threshold value ThA (Sa/Sb<ThA) (No atS114A), the search determination unit 122 determines that the search ofthe storage object 90 has not been completed (step S116A). On the otherhand, when the ratio Sa/Sb is equal to or larger than the thresholdvalue ThA (Sa/Sb≥ThA) (Yes at S114A), the search determining unit 122determines that the search of the storage object 90 has been completed(step S118A). Note that the amount of unmeasured areas and the number ofunrecognized target objects 80 can be reduced by having the sensor 12perform measurement in a viewpoint position(s) determined by thelater-described process in the step S130. Therefore, a possibility thatthe relation “Sa/Sb≥ThA” holds can be increased by having the sensor 12repeat measurements in viewpoint positions determined by the process inthe step S130.

FIG. 7 is a flowchart showing a second example of the search completiondetermination method performed by the search determination unit 122according to the first embodiment. Further, FIG. 8 is a diagram forexplaining the second example shown in FIG. 7. FIG. 8 is a plan viewshowing a state in which target objects 80A, 80B and 80C are disposed onthe shelf board 92 of the storage object 90 as viewed from above (asviewed in a Z-axis positive direction).

The search determination unit 122 extracts edges of the already-measuredplacement available area of the storage object 90 from the measurementdata (the 3D environmental information) (step S112B). Note that when theplacement available area corresponds to the shelf board 92, edges 92 eof the placement available area are boundaries between the shelf board92 and the wall surfaces 94 (indicated by bold lines in FIG. 8). Notethat since coordinate data of the shelf board 92 is already included inthe storage object information (the 3D environmental information), it iseasy to recognize which areas correspond to the edges 92 e in themeasurement data (the 3D environmental information).

The search determination unit 122 determines whether or not all theedges 92 e of the placement available area have been detected (stepS114B). When all the edges 92 e have not been detected (No at S114B),the search determination unit 122 determines that the search of thestorage object 90 has not been completed (step S116B). On the otherhand, when all the edges 92 e have been detected (Yes at S114B), thesearch determining unit 122 determines that the search of the storageobject 90 has been completed (step S118B). In the example shown in FIG.8, no edge 92 e is detected in parts that become blind spots due to thetarget objects 80 and parts located outside the field-of-view range 12 a(indicated by broken-line ellipses in FIG. 8). Therefore, in this case,the search determination unit 122 determines that the search of thestorage object 90 has not been completed. Note that the amount ofunmeasured edges 92 e can be reduced by having the sensor 12 performmeasurement in a viewpoint position(s) determined by the later-describedprocess in the step S130. Therefore, all the edges 92 e could bedetected by having the sensor 12 repeat measurements in viewpointpositions determined by the process in the step S130.

Note that even when the placement available area is a 3D space, theabove-described first and second examples can be applied. In the firstexample, the search determining unit 122 may compare the sum total ofthe volume of a measured space in the storage object 90 and the volumesof recognized target objects 80 with the volume of the placementavailable area in the storage object 90. In the second example, thesearch determination unit 122 may determine whether all the wallsurfaces 94 around the shelf board 92 have been measured.

The optimal viewpoint selection unit 124 determines whether or notviewpoint candidates have already been calculated (step S120). When theviewpoint candidates have not been calculated (No at S120), the optimalviewpoint selection unit 124 calculates viewpoint candidates (stepS122). On the other hand, when the viewpoint candidates have alreadybeen calculated (Yes at S120), the optimal viewpoint selection unit 124examines other viewpoint positions included in the viewpoint candidates(S142 to S150). Note that the process in the step S122 may be performedonly once, i.e., performed only in the first loop.

In the step S120, the optimal viewpoint selection unit 124 calculatesviewpoint positions of the sensor 12 from which the sensor 12 canmeasure the placement available area as viewpoint candidates by usingthe storage object information and the 3D environmental information. Inthis case, the optimal viewpoint selection unit 124 takes only thestorage object 90 into consideration and does not take the presence oftarget objects 80 disposed in the storage object 90 into consideration.Note that the process in the step S122 is performed in the 3D virtualspace represented by the 3D environmental information.

Specifically, the optimal viewpoint selection unit 124 calculates, forexample, a viewpoint position in which at least a part of the placementavailable area of the storage object 90 is included in the field-of-viewrange 12 a (the viewing angle) of the sensor 12 by using the storageobject information and the 3D environmental information. The optimalviewpoint selection unit 124 calculates, for example, a viewpointposition in which at least a part of the shelf board 92 of the storageobject 90 is included. For example, the optimal viewpoint selection unit124 determines whether or not an image of the shelf board 92 is includedin the field-of-view range 12 a (the viewing angle) of the sensor 12when the viewpoint position is moved in the 3D virtual space. Further,when the image of the shelf board 92 is included in the field-of-viewrange 12 a, the optimal viewpoint selection unit 124 defines thatviewpoint position as a viewpoint candidate.

FIGS. 9 to 11 are diagrams for explaining a viewpoint candidatecalculation method according to the first embodiment. In FIG. 9, atleast a part of the shelf board 92 is not shielded by the wall surface94 and is included in the field-of-view range 12 a at either ofviewpoints A and B. Therefore, the viewpoints A and B are included inthe viewpoint candidates. In contrast, at a viewpoint C, since the shelfboard 92 is entirely shielded by the wall surface 94, no area of theshelf board 92 is included in the field-of-view range 12 a. Therefore,the viewpoint C is not included in the viewpoint candidates.

In FIG. 10, a range of viewpoint candidates 12 b in which the sensor 12can be positioned is indicated by a bold-line arrow. The sensor 12 canmeasure at least a part of the shelf board 92 at any viewpoint in thisrange of viewpoint candidates 12 b. Meanwhile, no part of the shelfboard 92 can be measured in a range of viewpoints indicated by abroken-line arrow and hence this range of viewpoints is excluded fromthe viewpoint candidates.

In FIG. 11, a range of viewpoint candidates 12 b which represents arange of postures (orientations) of the sensor 12 at a position A isindicated by a bold-line arrow. The sensor 12 can measure the shelfboard 92 in at any viewpoint in this range of viewpoint candidates 12 b.Meanwhile, the shelf board 92 cannot be measured in a range ofviewpoints indicated by a broken-line arrow and hence this range ofviewpoints is excluded from the viewpoint candidates.

Next, the optimal viewpoint selection unit 124 selects an optimalviewpoint position, which is an optimal viewpoint position as the nextviewpoint position of the sensor 12, from the viewpoint candidates (stepS130). Specifically, the optimal viewpoint selection unit 124 selects,as the optimal viewpoint position, a position where the sensor 12 canmeasure a target object 80 (e.g., the target object 80C in FIG. 1) thatis partially shielded by an already-recognized target object(s) 80(e.g., the target object(s) 80A or/and 80B in FIG. 1). Further, theoptimal viewpoint selection unit 124 selects, as the optimal viewpointposition, a position where the sensor 12 can measure an area that isshielded by the already-recognized target object(s) 80 (e.g., the targetobject(s) 80A or/and 80B in FIG. 1). That is, the position determinationunit 120 determines, by using the 3D environmental information, aposition of the sensor 12 where the sensor 12 can take a distance imagein which a size of an area shielded by the already-recognized targetobject(s) 80 (e.g., the target object(s) 80A or/and 80B in FIG. 1) islarger as the optimal viewpoint position.

FIGS. 12 to 14 are diagrams for explaining processes performed by theoptimal viewpoint selection unit 124 according to the first embodiment.The examples shown in FIGS. 12 and 13 can be used to measure an object80 (e.g., the object 80C in FIG. 1) that has not been recognized becausea part of it is shielded, though another part of it has been measured.Further, the example shown in FIG. 14 can be used in a case where it isunknown whether or not a target object 80 is disposed in an area(s)corresponding to a blind spot(s) of a recognized target object(s) 80.The case of the example shown in FIG. 1 is described hereinafter.

In the example shown in FIG. 12, the optimal viewpoint selection unit124 selects, for example, a viewpoint position in which an unrecognizedtarget object 80 and target objects 80 located around this unrecognizedtarget object 80 are widely distributed (i.e., scattered over a widearea) in the field of view (the angle of view) of the sensor 12.Specifically, firstly, the optimal viewpoint selection unit 124 excludesposition data related to the storage object 90 from the 3D environmentalinformation as indicated by broken lines in FIG. 12. Then, in the 3Dvirtual space represented by the 3D environmental information, theoptimal viewpoint selection unit 124 extracts, for each of a pluralityof viewpoint positions imaginarily defined in the range of viewpointpositions calculated in the process in the step S122, a distance imagethat is obtained when the aforementioned plurality of target objects 80are viewed from that imaginarily-defined viewpoint position. Note thatthe optimal viewpoint selection unit 124 can generate a distance imagethat is obtained when the target objects 80 are viewed from animaginarily-defined viewpoint position by using sensor informationindicating an angle of view, etc. of the sensor 12.

Then, the optimal viewpoint selection unit 124 calculates, for each ofthe distance images, a distance between both ends of the plurality oftarget objects 80. The optimal viewpoint selection unit 124 selects aviewpoint position corresponding to a distance image having the longestdistance between both ends of the targets objects 80 as an optimalviewpoint position. Note that the optimal viewpoint selection unit 124may select a viewpoint position corresponding to a distance image inwhich the distance between both ends of the targets objects 80 isgreater than a predetermined threshold as the optimal viewpointposition. Note that “the distance between both ends” may not be anactual distance, but may be a distance in the distance image. Therefore,“the distance between both ends” may correspond to the number of pixelsin the distance image (This also applies to the example shown in FIG.13). Note that in a distance image in which “the distance between bothends” is larger may include a larger area that includes a part of theunrecognized target object 80.

In the example shown in FIG. 12, a distance image ImX including targetobject images 80AI, 80BI and 80CI, which are images of the targetobjects 80A, 80B and 80C, respectively, viewed from a viewpoint X, isextracted. Similarly, distance images ImY and ImZ that include targetobject images 80AI, 80BI and 80CI viewed from viewpoints Y and Z,respectively, are extracted. Then, the optimal viewpoint selection unit124 calculates a distance Wx between both ends of the target objects80A, 80B and 80C in the distance image ImX. Similarly, the optimalviewpoint selection unit 124 calculates distances Wy and Wz between bothends of the target objects 80A, 80B and 80C in the distance images ImYand ImZ, respectively. Then, the optimal viewpoint selection unit 124determines as follows: Wx>Wy; and Wx>Wz. That is, the optimal viewpointselection unit 124 determines that the distance image ImX has thelongest distance between both ends. Therefore, the optimal viewpointselection unit 124 selects the viewpoint X as an optimal viewpointposition. In this way, in the example shown in FIG. 12, the positiondetermination unit 120 determines, as the optimal viewpoint position, aposition of the sensor 12 where the sensor 12 can take a distance imagein which a size of an area (a part of the unrecognized target object80C) shielded by an already-recognized target object(s) 80 is larger.

Incidentally, when measurement has been made only from a viewpointcorresponding to the viewpoint Y, a part indicated by an arrow A1 inFIG. 12 has not been actually measured yet by the sensor 12 at the timeof this process. Therefore, since position data of the part indicated bythe arrow A1 is not included in the 3D environmental information, thispart may be missing in the distance images ImX and ImZ (This alsoapplies to FIG. 13). Further, when measurement has been made only from aviewpoint corresponding to the viewpoint Y, a part indicated by an arrowB1 has also not been measured yet. However, since the target objects 80Aand 80B have already been recognized, it is possible to draw the partindicated by the arrow B1 by using the target object information.

In the example shown in FIG. 13, the optimal viewpoint selection unit124 selects, for example, a viewpoint position in which a distancebetween the center of the unrecognized target object 80 and the centerof an already-recognized target object(s) 80 in the distance imagebecomes larger. Specifically, the optimal viewpoint selection unit 124calculates a central axis(es) of the recognized target object(s) 80 inthe 3D virtual space represented by the 3D environmental information byusing the target object information of the recognized target object 80.Note that the target object information may include informationindicating the central axis of the target object(s) 80 beforehand.

Further, the optimal viewpoint selection unit 124 calculates a centralaxis of the unrecognized target object 80 in the Z-axis direction byusing the 3D measurement data. Note that in the 3D measurement data,only a part of the unrecognized target object 80 has been measured.Therefore, the optimal viewpoint selection unit 124 calculates thecentral axis of the unrecognized target object 80 in the range that canbe estimated from the measured part. For example, when the top surfaceof the unrecognized target object 80 has already been measured, theoptimal viewpoint selection unit 124 may presume (i.e., regard) an axisthat passes through the center of gravity of the top surface as thecentral axis. Further, when the left and right side surfaces of theunrecognized target object 80 have already been measured, the optimalviewpoint selection unit 124 may presume (i.e., regard) an axis thatpasses through the middle between the left and right side surfaces asthe central axis.

Then, in the 3D virtual space represented by the 3D environmentalinformation, the optimal viewpoint selection unit 124 extracts, for eachof a plurality of viewpoint positions imaginarily defined in the rangeof viewpoint candidates calculated in the process in the step S122, adistance image that is obtained when the aforementioned plurality oftarget objects 80 are viewed from that imaginarily-defined viewpointposition. Then, the optimal viewpoint selection unit 124 calculates, foreach of the distance images, a distance between the central axis of theunrecognized target object 80 and the central axis(es) of thealready-recognized target object(s) 80. The optimal viewpoint selectionunit 124 selects a viewpoint position corresponding to a distance imagehaving the longest distance between the centers of these target objectsas an optimal viewpoint position. Note that the optimal viewpointselection unit 124 may select a viewpoint position corresponding to adistance image in which the distance between the centers of the targetobjects is greater than a predetermined threshold as the optimalviewpoint position. Note that in a distance image in which “the distancebetween the centers” is larger may include a larger area that includes apart of the unrecognized target object 80.

In the example shown in FIG. 13, distance images ImX and ImY thatinclude target object images 80AI, 80BI and 80CI as viewed fromviewpoints X and Y, respectively, are extracted. Then, the optimalviewpoint selection unit 124 calculates a distance Dx1 between a centralaxis 80Cc of the target object 80C and a central axis 80Ac of the targetobject 80A, and a distance Dx2 between the central axis 80Cc of thetarget object 80C and a central axis 80Bc of the target object 80B inthe range image ImX. Similarly, the optimal viewpoint selection unit 124calculates a distance Dy1 between a central axis 80Cc of the targetobject 80C and a central axis 80Ac of the target object 80A, and adistance Dy2 between the central axis 80Cc of the target object 80C anda central axis 80Bc of the target object 80B in the range image ImY.Then, for example, the optimal viewpoint selection unit 124 compares anaverage value (or a maximum value) of the inter-central-axis distancesDx1 and Dx2 in the distance image ImX with an average value (or amaximum value) of the inter-central-axis distances Dy1 and Dy2 in thedistance image ImY. Then, the optimal viewpoint selection unit 124determines that the distance image ImX has a larger average value (or alarger maximum value) and hence selects the viewpoint X as an optimalviewpoint position. In this way, in the example shown in FIG. 13, theposition determination unit 120 determines, as the optimal viewpointposition, a position of the sensor 12 where the sensor 12 can take adistance image in which a size of an area (a part of the unrecognizedtarget object 80C) shielded by an already-recognized target object(s) 80is larger.

Note that in the example shown in FIG. 13, only the distance between thecentral axes of the target objects 80 needs to be obtained. Therefore,in the example shown in FIG. 13, images of target objects 80 themselvesdo not need to be included in the extracted distance image. That is,only the central axis of each target object 80 needs to be included inthe extracted distance image.

In the example shown in FIG. 14, the optimal viewpoint selection unit124 selects, for example, a viewpoint position in which an unmeasuredarea in the placement available area of the storage object 90 isincluded in the field-of-view range 12 a as much as possible. Note thatthe unmeasured area means areas that were shielded by the targetobject(s) 80 or located outside the field-of-view range 12 a whenmeasurement was performed from an already-selected viewpointposition(s). FIG. 14 is a plan view showing a state in which targetobjects 80A, 80B and 80C are disposed on the shelf board 92 of thestorage object 90 as viewed from above (as viewed in a Z-axis positivedirection). Further, it is assumed that the target objects 80A and 80Bhave already been recognized.

The optimal viewpoint selection unit 124 extracts an unmeasured areafrom the placement available area of the storage object 90.Specifically, the optimal viewpoint selection unit 124 geometricallyextracts an area of the shelf board 92 that has not been measured (i.e.,an unmeasured area of the shelf board 92) from the 3D environmentalinformation, the 3D measurement data, and the storage objectinformation. More specifically, the optimal viewpoint selection unit 124extracts, as the unmeasured area, an area that is not indicated as beingalready-measured in the 3D measurement data (the 3D environmentalinformation) from position data of the top surface of the shelf board 92included in the storage object information.

Then, in the 3D virtual space represented by the 3D environmentalinformation, the optimal viewpoint selection unit 124 extracts, for eachof a plurality of viewpoint positions imaginarily defined in the rangeof viewpoint candidates, a distance image that is obtained when theshelf board 92 is viewed from that imaginarily-defined viewpointposition. Then, the optimal viewpoint selection unit 124 calculates, foreach of the distance images, a size of the unmeasured area of the shelfboard 92. The optimal viewpoint selection unit 124 selects a viewpointposition corresponding to a distance image having the largest size ofthe unmeasured area as an optimal viewpoint position. Note that “thesize of the unmeasured area” may not be an actual size, but may be asize in the distance image. Therefore, “the size of the unmeasured area”corresponds to the number of pixels in a part corresponding to theunmeasured area in the distance image.

In the example shown in FIG. 14, the optimal viewpoint selection unit124 determines that an area(s) that cannot be measured when the sensor12 performs measurement from a viewpoint position indicated by an arrowY (i.e., hatched areas in FIG. 14) will be included in the field-of-viewrange 12 a as much as possible when the sensor 12 performs measurementfrom a viewpoint position indicated by an arrow Z. Therefore, theoptimal viewpoint selection unit 124 selects the viewpoint positioncorresponding to the arrow Z as an optimal viewpoint position. In thisway, in the example shown in FIG. 14, the position determination unit120 determines, as the optimal viewpoint position, a position of thesensor 12 where the sensor 12 can take a distance image in which a sizeof an area shielded by an already-recognized target object(s) 80 islarger.

Note that in addition to the above-described examples shown in FIGS. 12to 14, the optimal viewpoint selection unit 124 may determine theoptimal viewpoint position based on discontinuity in measured positiondata of an unrecognized target object 80. Specifically, the optimalviewpoint selection unit 124 may determines, as the optimal viewpointposition, a position of the sensor 12 from which an image in which, inthe already-measured position data of the unrecognized target object 80,a part that became discontinuous due to an already-recognized targetobject 80, rather than the storage object 90, becomes larger is taken.Note that “the part that became discontinuous due to thealready-recognized target object 80” means an area that is shielded bythe already-recognized target object 80. Further, the optimal viewpointselection unit 124 may perform a process for selecting an optimalviewpoint position in which at least two of the above-described examplesshown in FIGS. 12 to 14 and the above-described other example arecombined.

The sensor control unit 140 controls the driving unit 14 and therebymoves the sensor 12 to the viewpoint position determined by the processin the step S130 (step S140). Then, the sensor control unit 140 controlsthe sensor 12 so as to perform measurement at the viewpoint position towhich the sensor 12 has moved, and acquires 3D measurement data (adistance image) from the sensor 12 (S100). Then, the object recognitionunit 114 performs the above-described object recognition process (S102).A possibility that an unrecognized target object 80 (e.g., the targetobject 80C) shown in FIG. 1, etc. can be recognized is increased byperforming measurement at the optimal viewpoint position.

Then, after the process in the step S104 is performed, the searchdetermination unit 122 determines whether or not the search of thestorage object 90 has been completed (S110), and determines whether ornot labeling has been made for all the target objects 80 (S112). Forexample, when the target object 80C is recognized and it is determinedthat no other target object 80 is disposed in the storage object 90 (Yesat S110 and Yes at S112), the control apparatus 100 determines that thedetection of all the target objects 80 disposed in the storage object 90has been completed and outputs a recognition result (S114). On the otherhand, when the target object 80C has not been recognized yet or when itis determined that it is unknown whether or not another target object 80is disposed in the storage object 90 (No at S110 or No at S112), thecontrol apparatus 100 proceeds to the process in the step S120.

Then, since the viewpoint candidates have already been calculated in theprocess in the step S120 (Yes at S120), the optimal viewpoint selectionunit 124 examines other viewpoint positions included in the viewpointcandidates (S142 to S150). Specifically, to prevent the same measurementfrom being performed again, the optimal viewpoint selection unit 124excludes the current viewpoint position, i.e., the viewpoint positionfrom which measurement has already been performed from the viewpointcandidates (step S142). Note that the optimal viewpoint selection unit124 may exclude the current viewpoint position and its surroundings fromthe viewpoint candidates. Then, the optimal viewpoint selection unit 124determines whether or not there is an unmeasured viewpoint position inthe viewpoint candidates (step S150). Specifically, the optimalviewpoint selection unit 124 determines whether or not there is aviewpoint position from which the sensor 12 can measure an area shieldedby the recognized target object 80 but has not performed measurement yet(i.e., an unmeasured viewpoint position).

When it is determined that there is an unmeasured viewpoint position (anunmeasured position) (Yes at S150), the optimal viewpoint selection unit124 performs the process in the step S130, i.e., selects an optimalviewpoint position. On the other hand, when it is determined that thereis no unmeasured viewpoint position (No at S150), the optimal viewpointselection unit 124 determines that further detection is impossible.Then, the control apparatus 100 outputs recognition results obtained upto this point to the interface unit 108 or the like (step S114).

FIG. 15 is a flowchart showing an example of a process for determiningwhether or not there is an unmeasured viewpoint position (S150)according to the first embodiment. Further, FIGS. 16 to 18 are diagramsfor explaining the process shown in FIG. 15. Firstly, the optimalviewpoint selection unit 124 extracts a shielded area from the placementavailable area (step S152). Note that the shielded area is an area inthe placement available area (e.g., an area on the upper surface of theshelf board 92) that is neither a measured area nor an area that isinferred to correspond to a bottom surface of an already-recognizedtarget object 80. That is, the shielded area is an area that is obtainedby excluding an area(s) that is inferred to correspond to the bottomsurface(s) of the already-recognized target object(s) 80 from thearea(s) that is shielded by an object(s) (i.e., the target object 80 orthe storage object 90) and has not been able to be measured. In FIGS. 16to 18, shielded areas 92 h are indicated by hatching.

Next, the optimal viewpoint selection unit 124 detects a straight linethat can extend from a boundary of the shielded area 92 h to an edge 92f located on an opened side of the placement available area withoutbeing obstructed by any obstacle (step S154). Note that the obstacle is,for example, a target object 80 disposed in the placement available area(the shelf board 92) or the wall surface 94. Both straight lines L1 andL2 shown in FIG. 16 can extend from the boundary of the shielding area92 h to the edge 92 f without being obstructed by any obstacle. Incontrast, both straight lines L3 and L4 shown in FIG. 17 are obstructedby (interfere with) an obstacle (the target object 80A or the targetobject 80B) when they extend from the boundary of the shielded area 92 hto the edge 92 f. Therefore, while the straight lines L1 and L2 aredetected, the straight lines L3 and L4 are not detected. Further, in theexample shown in FIG. 18, the boundary of the shielded area 92 h cannotbe connected with the edge 92 f by a straight line due to the targetsobjects 80D and 80E. When an object is disposed with a ratio higher thana predetermined threshold with respect to the horizontal direction (andthe vertical direction) of the placement available area as describedabove, the shielded area 92 h cannot be measured.

The optimal viewpoint selection unit 124 determines whether or not astraight line is detected in the process in the step S154 (step S156).When a straight line is detected (Yes at S156), the optimal viewpointselection unit 124 determines that there is an unmeasured viewpointposition (step S158). On the other hand, when no straight line isdetected (No at S156), the optimal viewpoint selection unit 124determines that there is no unmeasured viewpoint position (step S160).In the example shown in FIG. 16, since a straight line is detected, itis determined that there is an unmeasured viewpoint position. On theother hand, in the example shown in FIG. 18, since no straight line isdetected, it is determined that there is no unmeasured viewpointposition.

Note that the examples shown in FIGS. 15 to 17 correspond to the firstexample of the process in the step S110 shown in FIGS. 5 and 6. That is,in the examples shown in FIGS. 15 to 17, the determination on theunmeasured viewpoint position is made based on whether or not there isan unmeasured area in the shelf board 92. Further, the determination onthe unmeasured viewpoint position can also be made by a methodcorresponding to the second example of the process in the step S110shown in FIGS. 7 and 8. In this case, the optimal viewpoint selectionunit 124 may determine whether or not there is a straight line that canextend from an unmeasured edge 92 e of the shelf board 92 (whichcorresponds to the wall surface 94) to an edge 92 f on the opened sideof the shelf board 92 without being obstructed by any obstacle.

As described above, the control apparatus 100 according to the firstembodiment determines, when detecting a target object 80 by using thesensor 12, a viewpoint position from which the sensor 12 can measure anarea that becomes a blind spot due to an obstacle such as another targetobject 80 as an optimal viewpoint position for the next measurement.Therefore, it is possible to reduce the number of movements of thesensor 12 and the time required therefor when the target object 80 isdetected by using the sensor 12. Accordingly, the control apparatus 100according to the first embodiment can efficiently detect a target object80 even when the target object 80 is shielded by another object(s).

Second Embodiment

Next, a second embodiment is described. The second embodiment differsfrom the first embodiment because a target object 80 can be removed inthe second embodiment.

FIG. 19 shows an object detection system 1 according to the secondembodiment. Further, FIG. 20 is a block diagram showing a hardwareconfiguration of the object detection system 1 according to the secondembodiment. The object detection system 1 according to the secondembodiment includes an object detection apparatus 10 and a controlapparatus 100. The object detection apparatus 10 includes a sensor 12, adriving unit 14 that drives the sensor 12, and an arm 20. The arm 20 is,for example, a robot arm, and includes a plurality of joints 22 and anend effector 24 capable of grasping an object. The arm 20 can grasp atarget object 80 under the control of the control apparatus 100.

FIG. 21 is a functional block diagram showing a configuration of thecontrol apparatus 100 according to the second embodiment. The controlapparatus 100 according to the second embodiment includes an informationstorage unit 112, an object recognition unit 114, an informationgeneration unit 116, a position determination unit 120, and a sensorcontrol unit 140. Further, the control apparatus 100 according to thesecond embodiment includes a removal determination unit 210 and an armcontrol unit 220.

FIG. 22 is a flowchart showing an object detection method performed bythe control apparatus 100 according to the second embodiment. Note thatin FIG. 22, illustrations of processes substantially similar to those inFIG. 4 are omitted. When there is no unmeasured viewpoint (No at S150),the removal determination unit 210 determines which target object 80needs to be removed in order to measure an unmeasured area (a shieldedarea) in the best way (step S210). Specifically, the removaldetermination unit 210 determines one of already-recognized targetobjects 80 which provides a largest measurable area when removed (i.e.,an object to be removed). A specific determination method will bedescribed later. As described above, the control apparatus 100 accordingto the second embodiment can determine an object to be removed(hereinafter also referred to as a removal object). As a result, it ispossible to measure an unmeasured area by removing the removal object inthe subsequent processes, so that the control apparatus 100 according tothe second embodiment can make it possible to efficiently measure theunmeasured area.

Then, the arm control unit 220 controls the arm 20 so as to take out(i.e., remove) the removal object determined in the step S210 and movesit to a location other than the storage object 90 (step S230).Specifically, the arm control unit 220 detects a position of the removalobject (the target object 80) by using the 3D environmental information.The arm control unit 220 extracts positions of the storage object 90 andother target objects 80, and detects (i.e., determines) a trajectory ofthe arm 20 that does not interfere (i.e., collide) with the storageobject 90 and the other target objects 80. Then, the arm control unit220 controls the joints 22 so that the arm 20 moves along the detectedtrajectory. Then, when the end effector 24 reaches a position where itcan grasp the removal object, the arm control unit 220 controls the endeffector 24 so as to grasp the removal object. Then, the arm controlunit 220 controls the arm 20 so as to move the removal object to otherplaces. Note that when a target object 80 that has been searched for isfound as a result of the removal of the removal object, the arm controlunit 220 may control the arm 20 so as to grasp and remove that targetobject 80. As described above, a removal object can be automaticallyremoved in the second embodiment. Therefore, it is possible toefficiently measure an unmeasured area.

FIG. 23 is a flowchart showing an example of a method for determining aremoval object according to the second embodiment. Further, FIGS. 24 to28 are diagrams for explaining the method shown in FIG. 23. Firstly, theremoval determination unit 210 selects a candidate (a candidate removalobject) that can be removed by using the arm 20 from among recognizedtarget objects 80 (step S212). Specifically, the removal determinationunit 210 makes, for each of already-recognized target objects 80, atrajectory plan for the arm 20 by which that target object 80 can begrasped, and selects a candidate removal object that can be grasped andremoved. Here, the number of selected candidate removal objects isrepresented by n. In the example shown in FIG. 24, it is assumed thattarget objects 80A, 80B and 80C are candidate removal objects. That is,the number n is three (n=3). Further, it is assumed that a target object80X has not been recognized yet.

Next, the removal determining unit 210 calculates, for each candidateremoval object k, an area v_k that cannot be measured due to thatcandidate removal object k (i.e., an unmeasurable area v_k) (k is aninteger between 1 to n) in the current sensor position (step S214).Referring to FIG. 25, the removal determination unit 210 calculates anunmeasurable area 82A (v_1) that becomes a blind spot due to the targetobject 80A (k=1), which is one of the candidate removal objects.Similarly, the removal determination unit 210 calculates unmeasurableareas 82B (v_2) and 82C (v_3) that become blind spots due to the targetobjects 80B (k=2) and 80C (k=3), respectively, which are candidateremoval objects. Note that parts of the unmeasurable areas 82B and 82Coverlap each other.

Next, the removal determining unit 210 calculates an area V_K thatcannot be measured due to n−1 candidate removal objects (i.e., anunmeasurable area V_K) under an assumption that the candidate removalobject k has been removed (Step S216). FIG. 26 shows unmeasurable areas82B and 82C, which become blind spots due to the target objects 80B and80C under an assumption that the target object 80A has been removed. Anarea where at least one of the unmeasurable areas 82B and 82C is presentis an unmeasurable area V_1 under the assumption that the target object80A (k=1) has been removed. That is, the unmeasurable area V_1corresponds to a union of the unmeasurable areas 82B (v_2) and 82C(v_3).

FIG. 27 shows unmeasurable areas 82A and 82C, which become blind spotsdue to the target objects 80A and 80C under an assumption that thetarget object 80B has been removed. An area where at least one of theunmeasurable areas 82A and 82C is present is an unmeasurable area V_2under the assumption that the target object 80B (k=2) has been removed.That is, the unmeasurable area V_2 corresponds to a union of theunmeasurable areas 82A (v_1) and 82C (v_3).

FIG. 28 shows unmeasurable areas 82A and 82B, which become blind spotsdue to the target objects 80A and 80B under an assumption that thetarget object 80C has been removed. An area where at least one of theunmeasurable areas 82A and 82B is present is an unmeasurable area V_3under the assumption that the target object 80C (k=3) has been removed.That is, the unmeasurable area V_3 corresponds to a union of theunmeasurable areas 82A (v_1) and 82B (v_2).

Then, the removal determining unit 210 determines, as the removalobject, a candidate removal object k that can reduce the unmeasurablearea V_k the most when removed (step S218). That is, the removaldetermination unit 210 determines, as the target object 80 that providesthe largest measurable area when removed, a candidate removal objectthat had been assumed to be removed when the unmeasurable area V_k wasreduced the most. In the examples shown in FIGS. 24 to 28, theunmeasurable area V_2 shown in FIG. 27 is the smallest among theunmeasurable areas V_1 to V_3. Therefore, the removal determining unit210 determines that the target object 80B should be removed.

Modified Example

Note that the present disclosure is not limited to the above-describedembodiments and they can be modified as desired without departing fromthe scope and spirit of the disclosure. For example, the order of stepsin the flowchart shown in FIG. 4, etc. can be changed as desired.Further, one or more steps in the flowchart shown in FIG. 4, etc. may beomitted. Further, although the optimal viewpoint position indicates 3Dcoordinates and an orientation (an angle) in the 3D environment 4 in theabove-described embodiments, the present disclosure is not limited tosuch configurations. The optimal viewpoint position may indicate 3Dcoordinates in the 3D environment.

Further, although the target object 80 is placed on the shelf board 92in the above-described embodiments, the present disclosure is notlimited to such configurations. For example, the target object 80 may behooked on a hook provided (e.g., attached) on the wall surface 94.Further, the target object 80 does not need to be disposed (stored) inthe storage object 90. The target object 80 may be placed on the floorsurface in the 3D environment 4.

Further, although the sensor 12 is a range sensor (a 3D sensor) in theabove-described embodiments, the present disclosure is not limited tosuch configurations. The sensor 12 may be a two-dimensional (2D) sensor,provided that it can generate 3D environmental information. However, byusing a range sensor, it is possible to easily recognize a 3D positionof an object without performing complex image processing (such as edgedetection and pattern matching).

Further, although a target object 80 that is determined to be a removalobject is removed by the arm 20 in the above-described secondembodiment, the present disclosure is not limited to suchconfigurations. The control apparatus 100 may output information as towhich target object 80 is the removal object to, for example, theinterface unit 108. Then, a user may manually remove the target object80 whose information was output to the interface unit 108.

Further, in the above-described examples, the program can be stored andprovided to a computer using any type of non-transitory computerreadable media. Non-transitory computer readable media include any typeof tangible storage media. Examples of non-transitory computer readablemedia include magnetic storage media (such as floppy disks, magnetictapes, hard disk drives, etc.), optical magnetic storage media (e.g.magneto-optical disks), CD-ROM (compact disc read only memory), CD-R(compact disc recordable), CD-R/W (compact disc rewritable), andsemiconductor memories (such as mask ROM, PROM (programmable ROM), EPROM(erasable PROM), flash ROM, RAM (random access memory), etc.). Theprogram may be provided to a computer using any type of transitorycomputer readable media. Examples of transitory computer readable mediainclude electric signals, optical signals, and electromagnetic waves.Transitory computer readable media can provide the program to a computerthrough a wired communication line (e.g. electric wires, and opticalfibers) or a wireless communication line.

From the disclosure thus described, it will be obvious that theembodiments of the disclosure may be varied in many ways. Suchvariations are not to be regarded as a departure from the spirit andscope of the disclosure, and all such modifications as would be obviousto one skilled in the art are intended for inclusion within the scope ofthe following claims.

What is claimed is:
 1. A control apparatus configured to detect a targetobject to be detected by controlling a sensor configured to measuresurroundings in a three-dimensional (3D) environment, the controlapparatus comprising: hardware including at least one memory configuredto store a computer program and at least one processor configured toexecute the computer program; an object recognition unit, implemented bythe hardware, configured to recognize the target object present in the3D environment by using measurement data acquired from the sensor; aninformation generation unit, implemented by the hardware, configured togenerate 3D environmental information indicating each point on an objectpresent in the 3D environment by integrating a plurality of measurementdata; a position determination unit, implemented by the hardware,configured to determine whether or not a search of a placement availablearea, which is an area where the target object can be placed, has beencompleted by using the 3D environmental information, and when it isdetermined that the search has not been completed, perform a process fordetermining an optimal position of the sensor for performing a nextmeasurement; and a sensor control unit, implemented by the hardware,configured to move the sensor to the determined optimal position,wherein the position determination unit determines, by using the 3Denvironmental information, a position of the sensor where the sensor cantake an image in which a size of an area shielded by at least one firstobject is larger as the optimal position, the at least one first objectbeing the target object recognized by the object recognition unit. 2.The control apparatus according to claim 1, wherein the positiondetermination unit calculates, as a candidate for the optimal position,a position of the sensor where the sensor can measure the placementavailable area according to at least one of a position and a shape of astorage object in which the target object can be placed, and selects theoptimal position from the candidate.
 3. The control apparatus accordingto claim 2, wherein the position determination unit determines whetheror not, among candidate positions, there is an unmeasured position fromwhich the sensor can measure the area shielded by the at least one firstobject but has not performed measurement yet, and when it is determinedthat there is the unmeasured position, performs a process fordetermining the optimal position of the sensor for performing the nextmeasurement.
 4. The control apparatus according to claim 3, furthercomprising a removal determination unit configured to, when it isdetermined that there is no unmeasured position, determine the at leastone first object to be removed so that the area shielded by the at leastone first object can be measured.
 5. The control apparatus according toclaim 4, further comprising an arm control unit configured to control anarm so that the determined at least one first object is removed.
 6. Anobject detection system comprising: a sensor configured to measuresurroundings in a 3D environment; and a control apparatus configured todetect a target object to be detected by controlling the sensor, whereinthe control apparatus comprises: hardware, including at least one memoryconfigured to store a computer program and at least one processorconfigured to execute the computer program; an object recognition unit,implemented by the hardware, configured to recognize the target objectpresent in the 3D environment by using measurement data acquired fromthe sensor; an information generation unit, implemented by the hardware,configured to generate 3D environmental information indicating eachpoint on an object present in the 3D environment by integrating aplurality of measurement data; a position determination unit,implemented by the hardware, configured to whether or not a search of aplacement available area, which is an area where the target object canbe placed, has been completed by using the 3D environmental information,and when it is determined that the search has not been completed,perform a process for determining an optimal position of the sensor forperforming a next measurement; and a sensor control unit, implemented bythe hardware, configured to move the sensor to the determined optimalposition, wherein the position determination unit determines, by usingthe 3D environmental information, a position of the sensor where thesensor can take an image in which a size of an area shielded by at leastone first object is larger as the optimal position, the at least onefirst object being the target object recognized by the objectrecognition unit.
 7. An object detection method for detecting a targetobject to be detected by controlling a sensor configured to measuresurroundings in a 3D environment, comprising: recognizing the targetobject present in the 3D environment by using measurement data acquiredfrom the sensor; generating 3D environmental information indicating eachpoint on an object present in the 3D environment by integrating aplurality of measurement data; determining, by using the 3Denvironmental information, whether or not a search of a placementavailable area, which is an area where the target object can be placed,has been completed by using the 3D environmental information, and whenit is determined that the search has not been completed, performing aprocess for determining a position of the sensor where the sensor cantake an image in which a size of an area shielded by at least one firstobject is larger as an optimal position of the sensor for performing anext measurement, the at least one first object being the recognizedtarget object; and moving the sensor to the determined optimal position.8. A non-transitory computer readable medium storing a program forperforming an object detection method in which a target object to bedetected is detect by controlling a sensor configured to measuresurroundings in a 3D environment, the program being adapted to cause acomputer to perform: recognizing the target object present in the 3Denvironment by using measurement data acquired from the sensor;generating 3D environmental information indicating each point on anobject present in the 3D environment by integrating a plurality ofmeasurement data; determining, by using the 3D environmentalinformation, whether or not a search of a placement available area,which is an area where the target object can be placed, has beencompleted by using the 3D environmental information, and when it isdetermined that the search has not been completed, performing a processfor determining a position of the sensor where the sensor can take animage in which a size of an area shielded by at least one first objectis larger as an optimal position of the sensor for performing a nextmeasurement, the at least one first object being the recognized targetobject; and moving the sensor to the determined optimal position.